Differentiable Forward and Backward Fixed-Point Iteration Layers

Recently, several studies have proposed methods to utilize some classes of optimization problems in designing deep neural networks encode constraints that conventional layers cannot capture. However, these are still their infancy and require special treatments, such as the analysis Karush–Kuhn–Tucker (KKT) condition, derive backpropagation formula. In this paper, we propose a new formulation called fixed-point iteration (FPI) layer, which facilitates use more complicated operations networks. The backward FPI is motivated by recurrent (RBP) algorithm, also proposed. contrast RBP, layer yields gradient using small network module without explicitly calculating Jacobian. actual applications, both forward can be treated nodes computational graphs. All components our method implemented at high level abstraction, allows efficient higher-order differentiations on nodes. addition, present two practical methods, net (FPI_NN) descent (FPI_GD) whereby update single step based learnable cost function, respectively. FPI_NN intuitive simple, while FPI_GD used train energy function been studied recently. While RBP related not applied examples, experiments show successfully real-world image denoising, optical flow, multi-label classification.